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ABSTRACT 



This thesis was a study which deals with two basics concepts in human 
decision making. The first is the role of information relevance, specifically the 
adverse effects of irrelevant information on decision quality. The second key 
concept was cognitive feedback and its value for supporting decision making. The 
thesis was designed to research the effectiveness of cognitive feedback in reducing 
the adverse effects of irrelevant information. The experiment tested the Lens 
Model indices: achievement, consistency and matching in task conditions of high 
and low predictability. Subjects were divided into blocks which differed in the 
availability of cognitive feedback and predictability conditions. The results of the 
experiment showed the subjects performed better in all Lens Model indices in the 
cognitive feedback condition. Subjects also had superior performance across all 
Lens Model indices in the high predictability condition. This thesis was intended 
to contribute to the research in the subject of human decision making. The results 
were of importance in support of future design of decision support systems. 
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I. 



Introduction 



A. Research Context 

1 . Use of feedback on decision making 

Researchers in several fields have studied the effects 
of feedback on decision making. From the fields of psychology 
and psychiatry, research has studied the way in which 
cognitive feedback is used by an individual to understand 
his/her environment (Doherty and Balzer, 1988) . Researchers 
in the field of marketing have been interested in how "good" 
decision makers depend on the use of cognitive feedback 
(Jacoby, Mazursky, Troutman, and Kuss, 1984) . Research has 
shown that some forms of feedback can have an effect on the 
quality of decision making. From this research, and others, 
there have emerged several theories which support this thesis. 
These topics will be discussed in detail in Chapter II. 

2. Effects of irrelevant information in decision 
making 

The effect of irrelevant information on decision 
making has been a subject of continuing attention in the field 
of social and clinical psychology. Research has been directed 
at the effects of information relevance on decision making 
(Streufert S., 1973), reducing the effects of irrelevant 
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information on experienced decision makers (Gaeth and 
Shanteau, 1984) , and the perception of information relevance 
(Streufert and Streufert, 1970) . One well accepted theory 
concerning information relevance is that the presence of 
irrelevant information is detrimental to decision making. The 
concept of information relevance is of key importance to this 
thesis. Research and theories pertaining to information 
relevance will be discussed in Chapter II. 

The research proposed here addresses the two points 
made above - (a) given that the presence of irrelevant 
information has an effect on decision making, (b) can the use 
of feedback reduce the effects of irrelevant information. In 
this study we will examine the effects of irrelevant 
information in task environments characterized by different 
degrees of predictability. We will draw conclusions as to the 
effectiveness of feedback as a method of reducing the effects 
of irrelevant information. 

3 . Relevance to information systems research 

The relevance of this study to research to Information 
Systems (IS) is in the domain of Decision Support Systems 
(DSS) . With the rapid growth of demand for both specialized 
and off-the-shelf DSS, emphasis is being placed on the quality 
of design. DSS provide users with information in order to 
enhance the quality of their decisions. Not all information 
displayed to the user at one time is relevant. If the 
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decision maker is given the opportunity to study his/her own 
decision rules through the use of some form of cognitive 
feedback, he can improve his/her decision rules. Not all DSS 
of today provide CFB to the user. This research will help to 
determine if such a capability can improve the quality of the 
DSS . 

B . Problem domain 

In this section, we outline the elements of the problem 
domain, thereby constituting the basis for the research 
question - the notion of cognitive feedback as an aid to 
decision making in environments characterized by the presence 
of irrelevant information. 

1 . Irrelevant information 

The effects of irrelevant information on decision 
making have been studied extensively. The consensus from this 
research yields two basic premises that are essential to this 
study. First, the presence of irrelevant information is a 
detriment to decision making (Gaeth and Shanteau, 1984) . 
Second, the adverse effects of irrelevant information are 
greater in complex environments than in relatively simple 
environments (Streufert and Streufert, 1973) . 

2 . Cognitive feedback 

The term feedback describes an environment that 
returns some measure of the output of a system back to the 
system which produced the output. Feedback then allows a 
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person to compare his/her present state to an ideal state, to 
adjust itself in light of that comparison (Doherty and Balzer, 
1988) . Feedback can be provided in two forms - outcome and 
cognitive. Outcome feedback deals with the accuracy of a 
response. Cognitive feedback provides information as to how 
that response was generated. Cognitive feedback is the return 
of some measure of the output of a person' s cognitive 
processes, to help that person come to terms with his/her 
environment . 

Cognitive feedback can take three forms in an 
experimental setting: 

(a) information about the relationships between cues 
and criterion, i.e. information about the task: 

(b) information about relationships between cues and 
the person's inference, i.e. information about the person's 
cognitive state, sometimes referred to as insight; 

(c) information about relationships between 
cognitions and distal objects. This category comprises 
indices of "functional validity" information (Doherty and 
Balzar, 1988) . 

3 . Research question 

Given that the presence of irrelevant information 
adversely affects task performance, can the use of cognitive 
feedback improve the quality of decisions? 
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C. Organization of the thesis 

The discussion proceeds as follows. Chapter II reviews 
the research in the effects of irrelevant information and the 
use of feedback in task situations. Chapter III describes the 
experimental setting and chapter IV discusses the data 
analysis and results. Chapter V summarizes and draws 
conclusions . 
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II. 



RESEARCH 



A. The Research Question - A Conceptual Framework 

The research question deals with two basic theoretical 
areas: information relevance and cognitive feedback. The 
purpose of this chapter is to lay the groundwork in these two 
research areas, giving logical support for the hypotheses to 
be made later in Chapter III. Important to both of these 
research areas is the topic of human judgment. 

1 . Human Judgment 

Understanding human judgment is important in the 
context of this research. Changes in human judgment are what 
we hope to effect by manipulation of the dependent variables. 
A background discussion of human judgment theories is included 
in section A.l. The mathematical and experimental 
representation of human judgment is provided by the lens model 
(Dudycha and Naylor, 1966) . The lens model and issues 
surrounding linear models of human judgment will be discussed 
in section B.l. 

2 . Information Relevance 

Research in the area of information relevance is 
extensive (Streufert, 1973; Gaeth and Shanteau, 1981, 1984; 
Adelman, 1981) . It is commonly accepted that irrelevant 
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information serves as a detriment to decision making. The 
effects of information relevance vary under different 
circumstances. Section C will discuss the elements of 
information relevance as it pertains to this research. 

3 . The Research Problem 

The research problem can be framed by the following 
statements : 

(i) Individuals make models of decision processes. 

(ii) Irrelevant information serves as a detriment to 
successful development and use of a decision model. 

(iii) Cognitive feedback has been proven successful in 
improving decision making, by providing decision makers a 
better insight into their model development and usage 
strategy . 

(iv) So, given that cognitive feedback improves 
decision making, can it be useful in overcoming the 
detrimental effects of irrelevant information? 

B. Human Judgment 

According to Brehmer, knowledge is a relation between two 
systems. One system in this relationship is, for the purpose 
of studying human judgment, a person and the other is some 
portion of the environment. A person who is believed to 
"know" a great deal is someone who understands relationships 
with his/her environment better that someone who understands 
these relationships less. These relationships are believed to 
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be probabalistic . Thus, perfect knowledge is infeasible, 
only knowledge where the person has a high probability that 
what he believes he knows, is actually the case. 

It has been the focus of many researchers in experimental 
and clinical psychology to study the way in which humans seek 
relationships between themselves and their environment 
(Brunswik, Brehmer, Doherty and Balzer, et al) . Doherty and 
Balzer referred to knowledge representation as a complex set 
of relations called a policy. Policies are sampled from the 
stimuli and then analyzed for their relevance. The relevant 
components are then tried for their usefulness, either one by 
one or in some combination (Doherty and Balzer, 1980) . 

The use of linear models was first suggested by Brunswik 
(1944) for studies of perception. A linear model suggests 
that there is a probablistic and functional relationship 
between an individual and his environment, and that the 
functions can be described and measured. These methods are 
used to test a series of hypotheses about the nature of the 
judgment process, hypotheses about the nature of cue weights, 
function forms, combination rules, and predictability 
(Brehmer, 1979) . The name of the model developed by Brunswik 
was the Lens Model . 

1 . The Lens Model 

The Lens Model is a linear regression model developed 
by Brunswik for assessing the dynamics of human choice 
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behavior in a probablistic environment. The following text 
from Dudycha and Naylor describes well the components of the 
Lens Model : 

The three basic elements to the model are the cues or 
stimulus dimensions (X x .... X k ) , the correct response or 
answer (Y.) , and the observed response of the individual 
(Y,) . Any choice or decision situation (trial) must of 
necessity include these three elements . Given many such 
decision trials it is possible to determine the 
statistical relevance of a cue X ± as a predictor of the 
criterion Y. by computing the zero-order correlation r %1 
over trials. This true cue validity, when squared, can be 
interpreted as an index of the diagnostic power of that 
cue as a source of information for predicting the correct 
state of nature . 

Also from Dudycha and Naylor are the following values 
used in the model : 

r„ = the correlation between the true or observed 
criterion and the predicted criterion . 

r, = the index of subject consistency, or the degree 
to which the subject consistently utilized his strategy as 
defined by his multiple regression equation. 

r a = the correlation which denotes subject 
achievement . It measures the degree of agreement between the 
criterion values and responses of the subject over n 
observations . 

r B = the correlation between the two sets of n 
predicted values, which reflects the degree to which the 
regression (policy) equation of the subject "matches" the 
regression (optimal) equation of the ecology. 

r c . = the correlation which reflects the degree that 
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the optimal equation of the environment can predict the actual 
responses of the subject. 

r c . = the correlation which reflects the degree to 
which the policy equation of the subject can predict the true 
criterion. (The above variables and definitions taken from 
Dudycha and Naylor, 1966) . 

The Lens Model is an accepted method for interpreting 
subject responses in single or multiple cue probablistic 
learning environments with linear tasks. Tucker further 
developed the model for tasks in which the judgment is a 
linear additive function to yield the following relationship: 

r a = GR.R. 

R. represents the predictability of the environment, R. 
represents the consistency of the subject, and G is the 
correlation between the predictions derived from the linear 
model of the environment with the linear model of the subject 
(matching index) . Thus, r, is called the achievement index. 

C. Information Relevance 

Relevant information is viewed as any information to which 
a meaningful task-oriented response is possible (Streufert, 
1973) . Thus, information which is not useful in the 
generation of meaningful responses is irrelevant. In an 
experimental setting, as well as in nature, irrelevant 
information is often mixed with relevant information. 

In terms of the lens model, irrelevant information is a 
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cue which has a low r. ± . A cue with little or no diagnostic 
power (diagnostic power is described in the previous section) 
is mostly or entirely irrelevant. 

Study in information relevance has been extensive. Most 
data suggest that increases in irrelevant information decrease 



performance to 


criterion (Streufert, 


1973; 


Streufert 


and 


Streufert , 1971 ; 


Gaeth and Shanteau, 


1984; 


Ettenson 


and 


Shanteau, 1987; 


et al) . This premise 


is well 


accepted 


as a 



starting point for further research in the area of information 
relevance . 

Gaeth and Shanteau (1984) sought to discover if this 
influence extended to experienced decision makers. In their 
study, the decision makers were trained agricultural students. 
They were asked to describe soil texture. Irrelevant to this 
determination is moisture in the soil. One important result 
of the experiment is benefit of training in reducing the 
adverse effects of irrelevant information. If this influence 
extends to experienced decision makers, then one reasonable 
approach to improving judgmental skills would be to reduce the 
effects of irrelevant information (Gaeth and Shanteau, 1984) . 

1 . Effects of Irrelevant Information 

The effects of irrelevant information are known to 
vary in different task environments. For example, adding 
irrelevant information in simpler environments usually does 
not have as great an effect in producing decrements in task 
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performance (Streufert and Streufert, 1970) . A subject can 
perform well in a task environment where irrelevant 
information is present as long as he can sort out relevant and 
irrelevant information. In complex environments it is more 
difficult to differentiate between relevant and irrelevant 
information . 

Streufert (1973) performed a study demonstrating the 
decreases in performance levels previously believed to be the 
effect of load increases were actually a function of 
information relevance. 

2 . Effectiveness of Training 

Gaeth and Shanteau's research using training is of 
hallmark importance to research in information relevance. The 
study utilized experienced agricultural judges in a soil 
sampling task. In a pretest, irrelevant information was shown 
to influence the decisions of the judges . This pretest is of 
some consequence, for it shows that even experts are subject 
to the effects of information irrelevance. Subsequent 
training, in two different forms: lecture training and 

interactive training, was shown to improve accuracy and reduce 
the influence of irrelevance. Gaeth and Shanteau's research 
utilizing training is of hallmark importance as it is the 
first substantial work investigating the potential for 
reducing the adverse effects of irrelevant information. 
Additionally, it is of great significance that the study was 
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able to differentiate between improvements in accuracy and 
decreases in the effects of irrelevant information, proving 
that the two are mutually exclusive. A follow-up study showed 
that the benefit from training was shown to extend for a year 
after the training was held. 



D . Feedback 

1 . Outcome Feedback 

The procedure of informing the subject in an 
experiment of the correct value in a task situation (Y.) 
immediately after that subject produces a response (Y,) 
defines outcome feedback (OFB) (Doherty and Balzer, 
1988) . Research has attempted to correlate improvements in 

decision quality based of the use of OFB. It is commonly 
accepted that OFB alone is of little assistance to decision 
makers learning complex inference tasks. OFB can in some 
cases serve as a detriment to decision making because it 
encourages departure from linearity (Lindell, 1976) . This 
departure from linearity is especially likely when the task 
environment is less than complete predictability. 

Brehmer (1980) explains the reasons why OFB is not 

useful for enhancing decision making: 

Confirmation will, of course, not teach the subjects about 
the actual validity of their hypothesis; it will only tell 
which hypotheses work, although the reason why the 
hypotheses work may be very different from what the 
subject thinks. . . When we have to learn from outcomes, it 
may in fact, be almost impossible to discover that one 
really does not know anything. This is especially true 
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when the concepts are very complex in the sense that each 
instance contains many dimensions. 

One effect seen when OFB is used alone is when a 

subject is given OFB demonstrating that his response is 

incorrect, he will often abandon the policy which was 

employed to produce the response, even if it was the correct 

policy. Repeated failure to get the "right" answer may cause 

the subject to abandon the idea that there is a policy at all 

and he may resort to guessing (Doherty and Balzer, 1988) . 

Outcome feedback in a less than perfectly predictable 

task apparently acts to confuse subjects and lower their 

consistency. Achievement is also adversely affected, because 

of this lowered consistency (Schmitt, Coyle, and Saari, 1977) . 

It follows that outcome feedback in task environments of low 

predictability is not useful for policy revision. 

2 . Defining cognitive feedback 

Cognitive feedback is described as the return of some 
measure of the output of a person's cognitive processes, to 
help that person come to terms with the environment. 
Cognitive feedback provides the person with information 
describing the relationships: 

(a) between cues and the criterion (Task Information 



TI) ; 

(b) between cues and the person's inference, i.e. 
information about the person's policy (Cognitive Information; 
Cl) and/or 
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(c) between cognitions and the distal objects. This 
category comprises indices of functional validity (Doherty and 
Balzer, 1988) . The third category, above, is added to parallel 
the lens model accurately. In the lens model, TI includes the 
indices on the ecological side of the lens . Cl includes all 
information on the subjects side. Functional validity refers 
to all three measures r a , G, and C described in section B.l. 

In terms of the Lens Model, TI is represented by R. 
or R, 2 , ecological validities (earlier called r %i or diagnostic 
power, see section B.l) or function forms relating the 
criterion to the cues. Cl is represented by the values R. or 
R. 2 in the Lens Model as correlation indices of predictability 
of the subject (consistency) . Another form of Cl are the 
usage coefficients (e.g., r i# ; the subject's decision policy) 
and function forms relating the judgment to the cues. 

3 . Effectiveness of Cognitive Feedback 

Cognitive feedback has been used in numerous 
experiments to produce changes in decision making policy. 
Cognitively oriented feedback results in higher levels of 
achievement than outcome feedback over different task 
properties (Adelman, 1981) . Other studies have demonstrated 
the superiority of CFB to no information, as evidenced by 
policy change between blocks after providing CFB. Research 
has been conducted to determine which component of CFB (TI, 
Cl, or FVI ) is most effective in making change. The bulk of 
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research on the subject shows that TI alone is sufficient to 
facilitate change. There has been research investigating the 
use of Cl alone, and TI + Cl . There is not adequate research 
to make assumptions about differential effects. All lens 
model indices seem to be influenced by TI and TI + Cl with R. 
being more sensitive (Doherty and Balzer, 1988) . 

The use of CFB has been found to be related to task 
predictability. The use cognitive feedback under conditions 
of extremely high task congruence and predictability has not 
been shown to produce higher accuracy than outcome feedback 
alone. Schmitt et al . (1977) found that as task 
predictability decreased, achievement with cognitive feedback 
became significantly higher than that with outcome feedback 
(Adelman, 1981) . Predictability is varied in experimental 
settings to manipulate task complexity (Steinman, 1976) . 

One finding of double-systems studies pertains to the 
effects of task predictability on the cognitive systems of the 
subjects . Results obtained in a variety of circumstances show 
that the consistency of a cognitive system varies with the 
redictability of the task, the lower the consistency of the 
cognitive system (Brehmer 1979) . This effect is related to 
the theory that policy formation and utilization is 
probablistic . 
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E . Framing the Research Question 

The previous sections are valuable for demonstrating the 
need for and validity of research in the proposed area. In 
the section on irrelevant information it was shown that a 
measurable portion of difficulty experienced by decision 
makers in complex tasks was due to the effects of irrelevant 
information. Streufert went one step further in showing that 
similar losses in accuracy previously blamed on load were 
actually a result of irrelevant information. Some research 
has begun to discover means for recouping or avoiding these 
losses. Gaeth and Shanteau have had success using training as 
a means to this end. They further point out that research in 
irrelevant information is abundant, while research into 
correcting for it is overdue (Gaeth and Shanteau, 1981) . 

The use of cognitive feedback has been proven to be an 
effective means of improving performance in many different 
task oriented environments. The bulk of the research 
indicates that the more complex and predictable the task is, 
the more profound the benefit is when cognitive feedback is 
utilized. It is reasonable, then, to believe that it will 
also have a measurable effect on decision making in task 
environments subject to the presence of irrelevant 
information . 

These are the fundamental principles of the study. It is 
clear from these themes that not only is the theoretical 
foundation for the study valid, but there is evidence that the 
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results of the research will benefit all pertinent areas of 
study. The research question is further developed Chapter 

III, which describes the experimental methodology, and Chapter 

IV, which discussed the same research but in relation to the 
outcome of the experiment. 
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Ill . 



METHOD 



A. Scope 

This chapter describes the experimental design used in the 
research. The experiment was divided into three phases: 
training, experiment, and debriefing. The rationale for the 
training is explained in the previous chapter. Its results 
will be described in this chapter in section G. Section I 
discusses the debriefing phase, and section J is a summary of 
this chapter. 

B . Hypotheses 

The first hypothesis is discussed in terms of the lens 
model index for achievement. All hypotheses were formulated 
on the individual level. 

Hj: Subjects receiving CFB will attain higher decision 
quality than those receiving only OFB. 

H 2 : Subjects will perform better in task environments of 
high predictability than those in environments of low 
predictability . 

1 . Description of Hypotheses 

Based on arguments discussed previously in Chapter II, 
we postulate that performance will improve in the cognitive 
feedback condition. Performance in this sense means that the 



19 



subject's responses will more closely resemble the committee 
score. Similarly, the subject's policy will resemble the 
committee policy. 

Section C.l of Chapter II reviews Streufert's findings 
that the presence of irrelevant information is more 
detrimental to decision makers in environments of higher task 
difficulty. It is proposed in H 2 that this effect will also 
hold true to differences in task predictability. 

C. Experimental design 

The purpose of this section is to describe the 
experimental design of the experiment. The experiment has 
three phases: training phase, experimental phase, and 
debriefing . 

1 . Description of design 

The experimental design has two components - between 
subjects effects and within subjects effects. The design 
chosen to accomplish this was a 2 (OFB vs CFB) X 2 (High vs 
Low Predictability) X 3 (Block sequence) factorial design. 

The between-sub jects effect is the difference in 
performance between those subjects given cognitive feedback 
and those only given outcome feedback. Another between- 
subject effect is the difference in performance between those 
subjects in the high predictability task environment and those 
in the low predictability environment. The between subjects 
design is described by Figure 1 below. 
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Task 



Feedback 



CFB/HI 


CFB/LO 


OFB/HI 


OFB/LO 



Figure 1 



As well as determining if systematic differences exist 
among experimental conditions, this study also seeks to study 
the effect within each condition over time. This within- 
subjects design involves measurements that occur over time. 
Time of response will be measured for all subjects to see if 
learning was occurring with each subject. 

The within-sub jects effect of irrelevant information 
was operationalized by varying which cue (of three) was 
irrelevant for each block. The order that each subject 
receives the blocks of trials was then operationalized using 
a Latin Squares design following the procedure in given in 
Kirk (1982) . 



The following figure demonstrates the separation of 
within-sub jects . 

Group 1 Group 2 Group 3 
Task order: Cue 2 Cue 3 Cue 1 

Cue 1 Cue 2 Cue 3 

Cue 3 Cue 1 Cue 2 

Therefore, group 1 gets the order 2, 1, 3; group 2 gets the 
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order 3, 2, 1; and group 3 gets the order 1, 3, 2. For 
example, subjects in group 1 will receive a task sequence of 
the following order: in block 1, cue 2 was irrelevant; in 
block 2, cue 1 was irrelevant; and in block three, cue three 
was irrelevant. 

2 . Experimental setting 

The sessions were conducted in a closed room in the 
presence of the experimenter. The training and experimental 
phases were conducted on an IBM compatible personal computer. 
Brief instructions were provided by the experimenter at the 
beginning of the training phase. From that point, the subject 
used the instructions provided and on-screen help provided by 
the software to progress through the experiment. The software 
was written by the experimenters. The debriefing phase 
involved filling out a brief questionnaire attached to the 
instruction booklet. 

D. Subjects 

1 . Choice of subjects 

The experiment was conducted with sixty subjects at 
the Naval Postgraduate School. Each subject participated 
individually. Each participant was assigned to one of four 
cells of subjects . These four blocks are described in Figure 
1 . Subjects were assigned to one of four between-sub jects 
conditions and one of three within-sub jects conditions . 
Randomization was ensured in both of these assignment 
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procedures . 

Information of demographics and task specific factors 
that may impinge on the results of the experiment was 
collected. Demographic factors studied for interaction in the 
experiment include: age, sex, full-time work experience, 
familiarity with computers, and time lapse since undergraduate 
study. The task-specific factor was related to whether the 
subject had any previous experience in the task. 

2 . The use of students as subjects 

One limitation of the study centers around the use of 
students in a laboratory environment. All participants had a 
minimum of four years management experiment. The issue is the 
extent to which it is possible to make a reasonable comparison 
between the subjects of the experiment and real-life decision 
makers . 

Previous research using graduate students as 
surrogates for managers failed to find any significant 
difference between the two groups in making production 
scheduling decisions. It is reasonable to assume that 
graduate students are acceptable as representative of decision 
makers . 

It is difficult to claim external validity for 
laboratory studies . The bulk of the research conducted in 
support of the premises which frame the research question is 
conducted in a laboratory setting. The similarities between 
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other research in the area of irrelevant research and this 
research indicates that it is no less suitable for 
generalization to real-life. 

E . Task 

The following section describes in detail the experimental 
task (see Appendix) . Also described are the different task 
environments in which subjects groups performed. 

1 . Description 

The instruction booklet given to each subject varied 
based on the task. Subjects allowed to use cognitive feedback 
were given additional instructions describing the different 
types of feedback at their disposal and the function keys to 
access them. The debriefing questionnaire was also different 
for the two different types of subjects. The only difference 
in the debriefing questions pertained to how useful the 
subject found the types of feedback were in making their 
decision rules. 

The task presented to the subject was a candidate 
screening scenario. The subject was given a set of three cues 
representing three scores of a potential job applicant. The 
three scores ranged from 1 to 9. Each of the three scores 
represented an applicants score or rating on each of three 
variables or conditions: experience relevant to the position 
(variable 1) , general abilities test score (variable 2) , and 
interviewer rating of management abilities (variable 3) . 
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Based on these ratings the subject was then required to award 
an overall rating of these candidates on a scale of 1 to 9. 

In the training phase these sets of three cues were 
presented to the subject one set at a time. The subject gave 
his response and the committee score was then shown. In the 
experimental phase, the sets of cues were blocked into groups 
of 26 applicants. This allowed for the subject to see the 
effects of his policy over several cases. There was no 
irrelevant cue in the training phase. 

F . Feedback 

The use of cognitive feedback was provided to half of the 
subjects in the experiment. Feedback was explained to the 
subject and made available. It was given to the subject only 
on request, and was available in a variety of forms. The 
various types of feedback are described in the next sections . 

1 . Subject policy feedback 

The purpose of policy feedback is to allow the subject 
to see the policy he is using in order to refine and improve 
upon it. This type of feedback was presented to the subject 
in the form of a bar graph. Each cue was represented on the 
bar graph in a different color or shade (see Figures 2 and 3 
below) . The size of each shade was determined by the weight 
that the subject was placing on this variable in arriving at 
his overall score for the applicant. If the cue has negative 
value, i.e. a high score in a variable would lower the overall 
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score, it was shown in appropriate size for its negative 
weight, but below the positive values. 

In previous research conducted on representation of 
cognitive feedback, it was found that visual information was 
the most useful to the subject (Doherty and Balzer, 1988) . 
Hammond (1971) stressed the importance of giving the subject 
a picture of his cognitive processes. For this reason, the 
graphical representation shown in the above figure was used. 
Each of the three cues was clearly distinguishable, making the 
figure clear and easy to understand. 

Cue weights were calculated as follows: 

(a) Beta weights were calculated from a multiple 
regression of cue values and the subject's estimates. 

(b) The weights were then transformed to represent 
percentages of the sum of the squared weights. Thus, beta 
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values of 0.4, 0.8, and 0.2 were shown as 0.2, 0.75, and 0.05. 

(c) Transformed weights on the three cues were then 
displayed as a horizontal stacked bar, on a 1-100 scale (i.e., 
the transformed weights added up to 100) . 

This form of feedback is described as a type of 
Cognitive Information (Cl) in Chapter II. This type of 
feedback provides the subject with the function forms that 
relate his/her judgment to the cues. The other type of 
cognitive information discussed in Chapter II is consistency 
information . 

2 . Consistency feedback 

Consistency feedback gives the subject an indication 
as to how well he was conforming to his decision policy. It 
may not be inherently obvious that different sets of responses 
can generate the same policy, but one set of responses may be 
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much more tightly grouped around the committee scores. Thus, 
the consistency feedback will provide the subject an 
indication as to how consistent he was with his mental model. 
It was designed also to give some assistance in improving his 
consistency. Both of these goals were achieved by providing 
the subject with consistency scores on individual cases. In 
other words, the subject will be given the values that he 
would have generated if he conformed strictly to his decision 
rule . 

Consistency feedback is a form of CFB called cognitive 
information (Cl) by Doherty and Balzer (see section 2.4.) In 
terms of the lens model, consistency information is R, or R 2 , . 
It can be expressed in terms of the correlation indices of 
predictability of the subject. The use of consistency 

"scores" was chosen to make it easy for the subject to 
understand and use . 

Consistency scores were calculated as follows: 

(a) First beta weights were derived from a multiple 
regression of cue values and the subject's estimates. 

(b) Criterion values were computed by multiplying cue 
values with respective beta weights. 

3 . Committee decision policy 

The purpose of this type of feedback was to provide 
the subject with a reference to model his policy after. It 
was stated in the instructions that the purpose is to model 
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the decisions of the committee. This type of feedback was 
presented to the subject in the same manner that his own 
policy was presented, i.e. through the use of a bar graph. As 
before, each cue was given a section of the bar graph 
proportional to the weight that the committee places on that 
cue. In every set of cues, two of the three were weighted 
equally and the third was irrelevant. 

This type of feedback is called task information (TI) 
by Doherty and Balzer (see section 2.4) . It is accepted as 
the most useful in effecting a change in behavior. This 
information provides the subject the function forms relating 
the criterion to the cues . 

4 . Outcome feedback 

The purpose of this type of feedback was to provide 
the subject some indication as to his performance, also known 
as the committee score. This form of feedback was presented 
to the subject as a single number value. This value was 
calculated by summing the result of multiplying each cue by 
its beta weight. This type of feedback gives the subject an 
indication of his achievement. This value is denoted as R„ in 
the lens model. 

5 . Combined feedback 

The purpose of this feedback was to allow the subject 
to view two different types of feedback at one time. For 
example, the subject can ask to see the committee policy and 
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his own policy side by side. This was a convenience provided 
the subject to allow him the best conditions for refining his 
policy. The types of combined feedback allowed to the subject 
are: committee and subject policy rules, information on 
subject decision rule and consistency. 

6. Multiple requests for feedback 

The subject was allowed to request feedback many 
times. The only restraint on this was that once a subject has 
requested outcome feedback, he may no longer make revisions to 
those scores for which the committee score was given. 

G . Training 

1 . Basis for training 

The purpose of training in the experiment was ensure 
that all of the subjects begin the experiment with the same 
knowledge of the task. Due to the context of the experiment, 
some subjects may come into the experiment with some 
preconceived ideas about which variables should have greater 
weights in their policy, i.e. some may feel that the interview 
score is more important than job experience. In the training 
phase, all cues were weighted equally (there was no 
irrelevant cue) . At the end of the training phase, all 
subjects should have adequate knowledge of the task to perform 
well . 
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2 . Training procedure 

During the training phase, all subjects received the 
same set of data. Each subject was asked to respond to 70 
cases. The subject was required to give each candidate an 
overall score from 1 to 9 . This was followed by the actual 
committee score. The cues, subject response, and committee 
score would remain on the screen until the subject presses a 
key to bring on the next set. Thus, the subject could take 
as long as he needed for the training but could only view one 
case at a time. 

H. Experimental phase 

1 . Experimental procedure common to all subjects 

The experiment was conducted over three blocks of 
twenty-six cases each. The order in which these blocks were 
presented to the subject was randomized in order to discount 
the effect of order effects . Appendix A contains details of 
the instructions given to each of the subjects at the 
beginning and end of the experiment. Each block required the 
subject to perform a repetition of one type of task. Each 
subject would sit at the terminal and read the introductory 
screen explaining how to progress through the experiment. 
After pressing a key, the subject was presented with twenty- 
six sets of three cues. The screen was split down the middle 
so that thirteen cases appear on each side. Each cue 
represents a value as described in section C.l. The subject 
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was allowed to make an overall evaluation of the candidate and 



award a score between 1 and 9. The subject may use the arrow 
keys to move from one case to another. No requirements exist 
that require the subject to evaluate the cases in order. 

2 . Use of cognitive feedback 

After ten scores have been filled in by the subject, 
he can request one of the types of feedback as described in 
section 3.5. To view the options for feedback, the subject 
need only press the END key. If no feedback was required, the 
subject may return to the experiment by pressing the N key. 
The subject may request outcome feedback at any time, but may 
no longer change his scores after doing so. When all cases 
have been evaluated, the subject again presses the END key and 
then was led through a series of questions to ensure that he 
was through with the block and then the next block was 
presented. No time limit was placed on the subject. After 
completion of each block, the subject was required to fill out 
a short questionnaire. 

I . Debriefing 

The objective of the debriefing phase was to have the 
subject provide a description of his own policy and thought 
processes as he progressed through the experiment. 
Information provided in this section will give insight into 
how the subjects viewed different aspects of the experiment. 
The debriefing results will provide the experimenter 
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information concerning how clear all phases of the experiment 
were to the subjects. 

J . Summary 

This chapter explained the experimental environment. Of 
special interest in this chapter are the following points: 

(1) The experiment included four different groups of fifteen 
subjects each. Within each of these groups, the subjects were 
further differentiated by the order in which the blocks were 
presented. This is to discount order effects, should one 
block of data be significantly different from another. 

(2) The first phase of the experiment was the training phase. 
The purpose of this phase is to ensure that all subjects enter 
the experiment with equivalent knowledge of the task and no 
prejudices about the simulated task were carried into the 
experiment. The training consisted of seventy-two trials. 

(3) Cognitive feedback was provided (for half the subjects) 
in three different forms: subject policy information, subject 
consistency information, and committee policy information. 

(4) Subjects all performed the same task in the simulation. 
They evaluated potential job candidates based on three values: 
work experience, test scores, and interview score. Outcome 
feedback was provided for all subjects upon request. 
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IV. 



RESULTS 



A. Statistical Modal 

The following analysis of variance (ANOVA) model suited 
for multiple Latin Squares was used to test hypothesis: 

Y ijk(i> m = (i + a± + fij + r k + x x + o m + <rx) kl + 

(Ta) kl + <nj) kJ + e ljk(1)B where: 

|1 is constant, 

a ± is the sequence of experimental tasks (see Chapter III, 
P • 21) , 

is the order of the task (j = 1, . . .3) , 

T k is the feedback condition, 

x x is the experimental task (i.e., cue 1 irrelevant, cue 2 
irrelevant, etc . ) , 

O m is the experimental participant (or subject, m = 1, . . .,53), 
and 

e ijMi)m t ^ ie experimental error term. 

This model was run for lens model indices achievement 
(R a ) , consistency (R.) , and matching (G) , and number of 

iterations (ITER) performed by the subject within a block. 
The analyses were conducted on SAS R statistical software, 
using the General Linear Models procedure. The general linear 
models procedure was chosen because the number of subjects 
assigned to each condition varied. The assignments of 
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subjects to blocks was as follows: CFB/H = 12, CFB/L = 14, 

OFB/H = 13, OFB/L = 12. 

B. Results 

1 . Training 

The results of the training phase are summarized in 
table 1 . The same dependent variables were analyzed in the 
training phase data. Different sets of cues were given to the 
subjects in the training phase based on assignment to task 
predictability conditions (i.e. all subjects in the high 
predictability conditions received the same cues in the 
training phase) . All subjects received one block of decisions 
during the training phase. The only significant effects seen 
in the training data was a significant variance between groups 
(high/low predictability) in the training phase. Analysis of 
training phase data is displayed in Table 1. 

2 . Experimental Results 

Tables 2—6 summarize the means and standard 
deviations for all of the independent variables for the 
experimental phase. Graphical representation of this data is 
provided for each set of data in the tables . 

a. Means and Standard Deviations 
Achievement 

Achievement is the measure of the agreement between 
the criterion values and responses of the subject over n 
observations. An inspection of the means for achievement (RA) 
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Table 1 



TRAINING DATA ANALYSIS 





Class Levels 


Values 








GROUP 4 


CFB/H CFB/L OFB/H 


OFB/L 




SEQ 3 


12 3 










Number of observations in data set - 


■ 53 


Dependent 


Variable : 


RA 












Sum of 


Mean 






Source 


DF 


Squares 


Square F 


Value 


Pr > F 


Model 


11 


0.490 


0.0446 


4.07 


0.0005 


Error 


41 


0.449 


0.0109 






Crctd Total 52 


0.939 








GROUP 


3 


0.4096 


0.1365 


12.48 


0.0001* 


SEQ 


2 


0.0123 


0.0062 


0.56 


0.5733 


GROUP *SEQ 


6 


0.0682 


0.0113 


1.04 


0.4141 


Dependent 


Variable : 


G 








Model 


11 


0.010 


0.0009 


0.79 


0.6478 


Error 


41 


0.048 


0.0012 






Crctd Total 52 


0.058 








GROUP 


3 


0.005 


0.0017 


1.50 


0.2277 


SEQ 


2 


0.0017 


0.0008 


0.72 


0.4949 


GROUP* SEQ 


6 


0.0032 


0.0005 


0.46 


0.8345 


Dependent Variable: 


RS 








Model 


11 


0.0918 


0.0080 


0.97 


0.4896 


Error 


41 


0.3537 


0.0090 






Crctd Total 52 


0.4454 








GROUP 


3 


0.0321 


0.0107 


1.24 


0.3064 


SEQ 


2 


0.0130 


0.0065 


0.76 


0.4763 


GROUP* SEQ 


6 


0.0466 


0.0078 


0.90 


0.5036 


* - Significant at 0.05 


level 






Table 2 


MEANS 


AND STANDARD DEVIATIONS 


FOR ACHIEVEMENT 




Group 




Block 1 


Block 


2 


Block 3 


CFB/H 




. 8 41 ( . 10 6) 


.818 (. 


192) 


. 864 ( . 131) 


CFB/L 




. 576 ( . 207) 


. 607 ( . 


,137) 


. 609 ( . 149) 


OFB/H 




.768 (.117) 


.744 (. 


081) 


. 709 ( .105) 


OFB/L 




. 502 ( . 1 69) 


. 470 ( . 


231) 


. 457 ( . 183) 



* Mean (St . Dev. ) 



in Table 2 show that subjects in the CFB/H condition had the 
highest degree of accuracy. The graphical representation of 



36 



this data, in Figure 4, shows that achievement in the 
cognitive feedback conditions improved in consecutive blocks 
while achievement declined in both outcome feedback 
conditions. This result supports H 2 as discussed in Chapter 
III. From the graph and the table it is also clear that for 
subjects within the same feedback condition, those in the high 
predictability condition had higher values for achievement 
than those in the low predictability condition. This supports 
H 2 as discussed in Chapter III. 




CorisJjstencx 

Consistency is defined as the multiple correlation 
between the cues and the criterion. Consistency for a subject 
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is calculated from estimates made by the subject in his/her 
final iteration in a given block. The values for consistency 
in Table 3 indicate that subjects in the CFB conditions were 
the most consistent in the utilization of their decision 
rules, as defined by the multiple regression equation. The 
graphical representation of this data, in Figure 5, shows that 
consistency improved in subsequent blocks in the cognitive 
feedback condition. 

Table 3 MEANS AND STANDARD DEVIATIONS FOR CONSISTENCY 



Group 



Block 1 Block 2 



Block 3 



CFB/H . 914 ( . 086) .911(.077) .944(.032) 

CFB/L . 914 ( . 046) .885(.107) .923(.053) 

OFB/H . 898 ( . 045) .886(.049) .866(.073) 

OFB/L . 8 67 ( . 101 ) .839(.105) .840(.117) 



* Mean (St . Dev . ) 



The highest mean for consistency was found in the 
CFB/H Block 3 condition, while the lowest was found in the 
OFB/L Blocks 2 and 3 condition. Subjects in the CFB condition 
tended to become extremely consistent in the final block, with 
very little variation, as indicated by the small standard 
deviation . 

Matching 

Matching is defined as the correlation between model 
estimates of the subject with the model of the environment 
(committee policy) . The values for matching in Table 4 
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Mean Score 



Consistency 




CFB/H — t— CFB/L OFB/H OFB/L 

Figure 5 Experimental results for consistency 
indicate that subjects in the high predictability had the 
greatest success in matching the committee policy with their 
own policy. Graphical representation of this data, in Figure 
6, shows that matching was best with cognitive feedback and 
high predictability conditions. The higher degree of matching 
between subject and committee in the cognitive feedback 
condition support H x as suggested in Chapter III. 

For cognitive feedback - low predictability 
conditions, matching improved with each subsequent block. It 
should be noted that although the matching index was on the 
average higher in the OFB/H condition than in the CFB/L 
condition, in the third block subjects in the CFB/L condition 
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Matching 



Mean Score 




— CFB/H —I— CFB/L OFB/H “B- OFB/L 

Figure 6 Experimental results for matching 

Table 4 MEANS AND STANDARD DEVIATIONS FOR MATCHING 



Group 



Block 1 



Block 2 Block 3 



CFB/H 

CFB/L 

OFB/H 

OFB/L 



.967 (.040) 
. 809 ( . 201) 
.877 ( . 148) 
.722 ( . 181) 



. 917 ( . 032) 
. 832 ( . 1 62) 
.878 (.075) 
. 692 ( . 273) 



. 948 (.120) 
. 870 ( . 157) 
. 8 69 ( . Ill ) 
.793 (.099) 



* — Mean (St . Dev . ) 



were able to surpass the matching ability of subjects in any 
of the OFB conditions. 

Iteratio n 

Iteration is described as the number of times that a 
subject requests feedback in a given block. The data in table 
V indicates that subjects in the cognitive feedback condition 
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ITERATIONS 



Mean Score 




— CFB/H “I— CFB/L OFB/H OFB/L 

Figure 7 Experimental results for iteration 
completed each block in the fewest number of iterations. This 
observation does not indicate that the subjects in the 
cognitive feedback conditions completed the blocks in the 
shortest amount of time. The graphical representation of this 
data is in Figure 7 . 

Table 5 MEANS AND STANDARD DEVIATIONS FOR ITERATION 



Group 



Block. 1 Block. 2 



Block 3 



CFB/H 

CFB/L 

OFB/H 

OFB/L 



11.9(7.90) 

9.29(7.18) 

8.23(7.63) 

13.29.86) 



8.21(4.92) 

6.86(4.00) 

15.9(8.97) 

18.3(10.4) 



7.93 (3.71) 
7.79(5.54) 
14.5(7.37) 
13.5(9.28) 



* Mean (St . Dev . ) 
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DIFFERENCE 



Mean Score 




— 1 — CFB/H CFB/L OFB/H OFB/L 

• b®tw®®n ®ub[®ct end committee weigh!® 

Figure 8 Experimental results for difference 

Differen ce 

Difference is described as the difference between the 
weights used by the subject and the committee weights. Thus, 
a small value for DIFF would indicate that the subject 
utilized similar cue weights as the committee. This value is 
smallest for subjects in the cognitive feedback condition. 
Graphical representation of this data is found in Figure 5. 

The means for achievement indicate that the main 
effects are due to groups (task condition) and subject 
(variations from one subject to another) . The same is true 
for consistency and matching. 
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Table 6 MEANS AND STANDARD DEVIAITONS FOR DIFFERENCE 



Group 



Block 1 Block 2 



Block 3 



CFB/H 

CFB/L 

OFB/H 

OFB/L 



. 497 ( . 193) . 5 63 ( . 

. 666 ( . 292) . 613 ( . 

.7 67 ( . 311 ) .75 8 ( . 

. 834 ( . 277) . 863 ( . 



287) 


.524 (.250) 


314) 


. 531 ( . 263) 


209) 


. 7 85 ( . 278 ) 


413) 


1.14 (1.29) 



* - Mean (St .Dev. ) 



Jb. ANOVA Results 

The ANOVA was conducted on the same variables in 
the previous section, namely: RA, RS, ITER, G, and DIFF. The 

results of the ANOVA are summarized in tables 7-9. 

Table 7 ANOVA RESULTS FOR ACHIEVEMENT 



Sum of 



Source of Variation 


DF 


Squares 


F Value 


Pr > F 


R-Square 


Model 


62 


4.75825204 


3.95 


0.0001* 


0.718280 


GROUP 


3 


2.99613739 


51.37 


0.0001* 




BLOCK 


2 


0.00370189 


0.10 


0.9093 




SEQ 


2 


0.08812805 


2.27 


0.1092 




TASK 


2 


0.03695659 


0.95 


0.3901 




LNAME 


47 


1.57564399 


1.72 


0.0125* 




GROUP *BLOCK 


6 


0.05768413 


0.49 


0.8110 





* - Significant 0.05 level 



The ANOVA for accuracy (RA) showed significant main 
effects for group (p, 0.0001) and participant (p<0.05) . 
Variances in accuracy not found to be significant were between 
blocks, sequence, task, and group-block. 

ANOVA for matching (G) showed that the variance of 
matching was significant in the same categories as 
consistency, groups (0.0001) and last name (0.0196) . Matching 
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did not vary significantly in any other conditions tested. 



Table 8 ANOVA RESULTS FOR ACHIEVEMENT 



Sum of 



Source of Variation 


DF 


Squares 


F Value 


Pr > F 


R-Square 


Model 


62 


0.66091699 


2.84 


0.0001* 


0.647277 


GROUP 


3 


0.11970669 


10.64 


0.0001* 




BLOCK 


2 


0.00897107 


1.20 


0.3070 




SEQ 


2 


0.00463097 


0.62 


0.5416 




TASK 


2 


0.03546305 


4.73 


0.0110* 




LNAME 


47 


0.46653530 


2.65 


0.0001* 




GROUP * BLOCK 


6 


0.02560992 


1.14 


0.3465 





* - Significant at 0.05 level 



ANOVA for consistency showed that variance was 
significant between groups (0.0001), task (0.011), and last 
name (0.0001) Consistency did not vary significantly between 
blocks, sequence, or group-block. 

Table 9 ANOVA RESULTS FOR MATCHING 



Sum of 



Source of Variation 


DF 


Squares 


F Value 


Pr > F 


R-Square 


Model 


62 


2.68665623 


2.20 


0.0002* 


0.587311 


GROUP 


3 


0.87393415 


14.81 


0.0001* 




BLOCK 


2 


0.04132201 


1.05 


0.3537 




SEQ 


2 


0.09566424 


2.43 


0.0932 




TASK 


2 


0.08119316 


2.06 


0.1325 




LNAME 


47 


1.52604186 


1.65 


0.0196* 




GROUP* BLOCK 


6 


0.06850081 


0.58 


0.7450 





* - Significant at 0.05 level 



c . Scheffe ' a Test 

Scheffe' s Test is a posterior test performed to 
determine if differences within a variable are significant 
enough to account for observed differences in experimental 
results. Scheffe's test was performed between the following 
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variables and all possible combination of task conditions: 
RA, G, RS, ITER. 

In achievement, Scheffe's Test indicated that the 
means were significantly different from each other (p < 0.05) 
between all task conditions. In matching, Scheffe's Test 
indicated that means were significantly different (p < 0.05) 
in all conditions except when compared between CFB/H - OFB/H 
and CFB - OFB/H. In consistency, all means were significantly 
different at the same level except when compared between CFB/H 
- CFB/L, CFB/L - OFB/H, and OFB/H - OFB/L . 
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V. 



CONCLUSIONS 



A . Scope 

The purpose of this chapter is to revisit the research 
topics discussed in earlier chapters in light of the results 
of this research. The limitations of this research and the 
potential impact on potential research will be included in 
this chapter. This chapter also discusses the conclusions of 
the research in terms of the majors areas of study with which 
it deals: information relevance and cognitive feedback; and 

limitations of this research. The chapter ends with a 
discussion of the relevance of this thesis to research in 
computer systems . 

B. Conclusions 

1 . Previous Research 

We argued previously that while research in 

information relevance was abundant, research in finding means 
of reducing the effects of irrelevant information is an 
important next step. In addition to this research, an example 
of such an effort is the research done by Gaeth and Shanteau 
( 1984 ) . 

The results of the research performed by Gaeth and 
Shanteau is particularly of importance in terms of improving 
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decision quality in environments where irrelevant information 
is present. In their research, training was found to be an 
effective measure in improving experienced decision makers 
decision quality, by reducing the effects of irrelevant 
information. Similarly, in this research, the use of 
cognitive feedback was shown to be effective as well in 
improving decision quality when irrelevant information was 
present . 

2 . Research Question 

The question posed by this study was: given that 

irrelevant information serves as a detriment to decision 
quality, can the use of cognitive feedback reduce the adverse 
effects of this detriment. 

C. Findings of the study 

Lens Model Indices 

Decision quality was operationalized in terms of the 
following three lens model indices: achievement, consistency, 

and matching. 

• Achievement : Achievement was shown to be the highest in 

conditions where subjects in the high predictability task 
environment received cognitive feedback. Achievement 
improved in subsequent blocks in the cognitive feedback 
condition while declining in the outcome feedback 
condition . 

• Matching : Subjects receiving cognitive feedback recorded 

higher matching index than those receiving only outcome 
feedback . 

• Consistency : Consistency was highest for subjects in the 

cognitive feedback conditions. Further, for subjects in 
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the cognitive feedback condition consistency improved in 
subsequent blocks . 

D . Application of Cognitive Feedback 

Brehmer has been a major contributor to research 
concerning cognitive feedback. His work emphasizes the 
importance of cognitive feedback to the point of dismissing 
outcome feedback entirely. Only cognitive feedback can give 
a person insight into relations between variables. Further, 
only cognitive feedback is of use in describing subject 
consistency . 

In support of Brehmer, Doherty and Balzer state that the 
human learning process is imperfect - "we need help" (Doherty 
and Balzer, 1988) Their research also points out the strong 
link between consistency and cognitive feedback. 

Cognitive feedback can be operationalized in three forms 
(discussed in chapter two) : task information, cognitive 
information, and functional validity information. Various 
research on cognitive feedback has stressed each of these as 
important in different instances. While there has been some 
agreement in the research, it is clear that future research 
should focus on the distinctions between these forms of 
cognitive feedback and the importance of each. 

The use of technology in the application of cognitive 
feedback is an exciting enterprise with potential for enormous 
growth. Personal decision support systems designed to assist 
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the user in decisions such as major purchases or solving 
interpersonal problems are within the realm of possibility in 
the near future. Providing the user with insight into his or 
her decision processes, with the promise of improving decision 
quality, is appealing to designers in a variety of areas. 

E. Limitations and Recommendations for Further Research 

It is in the best interest of the experimenter to control 
and limit the scope an experiment in order to reduce the 
chance of numerous interaction effects or effects which cannot 
be explained by manipulation of the dependent variables. The 
loss associated with the gains of control of scope is that 
only a small part of a large picture may be examined at one 
time. The issues addressed in this section are the rest of 
the picture - those issues which bear some significance to the 
research topics but extend beyond the scope of this research 
1 . Information Relevance 

From previous research on information relevance is 
clear as to the adverse effects associated with irrelevant 
information. Other aspects of information relevance are less 
clear from the research completed to date. Further research 
should direct emphasis upon the following points: 

• Given that subjects in more predictable task environments 
are less affected by irrelevant information, where and why 
is this distinction seen? Why is cognitive feedback more 
effective in more predictable environments. 

• How does an individual determine which information is 
relevant and which is not? Do methods for sorting 
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information based on relevance increase in complexity as 
the information itself increases in complexity? 

• Overcoming the adverse effects of irrelevant information 
in decision making tasks. 

• The impact of training in improving task performance in 
task environments where irrelevant information is present. 

2 . Cognitive Feedback 

One limitation to experimental research on cognitions 
and policy formation is that an experimental setting leads 
subjects to form policies and cognitions differently than in 
the real world. One example of such is knowledge acquisition. 
In this research, for example, the subject is given three cues 
from which he must make a decision. The subject has a priori 
knowledge that he will be given sets of cues. He is not 
expected to do anything to get this information. This aspect 
of the experimental setting is different to the way in which 
individuals gain knowledge in the real world. In other words, 
an experimental subject may be led to know that he or she is 
expected to form a policy, and may even be led as to what form 
of policy is expected. 

Another limitation of the study of human cognitive 
processes stems from the use of the Lens Model . The Lens 
Model, while an accepted model for studying human behavior, is 
rigid and limits the results to the confines of a finite 
number of coefficients. 

Cognitive feedback studies thus far (including this 
research) have not explored how decision makers use cognitive 
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feedback (perhaps because of the difficulty foreseen in 
operationalization) . For example, when and why does a 
decision maker request feedback? Is feedback used to confirm 
an idea more effective than feedback used initially? What 
type of decision makers benefit the most from the different 
types of feedback? 

An area for further research is the use of cognitve 
feedback in conjunction with training to assist people to 
detect irrelevant information. The results of the work done 
with training discussed in this thesis indicates that there is 
great promise in this venture. 

F . Relevance to Research in Computer Systems 

One main thrust of research in computer systems is in 
decision support systems. The technology in decision support 
systems has reached a point where the competition is no longer 
between the capabilities of the hardware or the power of the 
coding. The limiting factor in most interactive software is 
the user. Thus, the area of the most potential for growth or 
additional understanding is in putting more power in the hands 
of the user. 

Conversely, power can be taken from the hands of the user 
of a decision support system through the presence of 
irrelevant information (supported by this research) . The use 
of cognitive feedback shows great promise in the field of 
decision support systems. Fundamental to this is a greater 
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understanding of the human cognitive process. 



6 . Summary 

The intent of this research has been to study decision 
making across different task situations. Through the 
employment of cognitive feedback, this study has shown that 
human decision making processes may be enhanced by simple 
decision aids. This result contributes to the knowledge base 
which deals with cognitive processes and decision making. A 
great deal is still unknown about the way humans make 
decisions. However, from this research and others, advances 
can be made in design technology for computer based decision 
tools . 
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APPENDIX 



The following appendix is the instruction set given to 
subject to guide them in completion of the experiment. The 
instruction set contained additional information for subjects 
in the cognitive feedback conditions. This additional 
information will be in bold. 



53 



